The present applicant has previously proposed class classification adaptation processing as image processing for improving image quality and so forth of images and for performing other image conversion, for example.
Class classification adaptation processing is made up of class classification processing and adaptation processing, wherein data of images is classified according to the nature thereof by the class classification processing, with adaptation processing begin performed for each class, adaptation processing being a technique as follows.
That is to say, with adaptation processing, for example, a low-image quality or standard image quality image (hereafter referred to as SD (Standard Definition) image as appropriate) is converted into a high-quality image (hereafter referred to as HD (High Definition) image as appropriate), by mapping (mapping) using a predetermined tap coefficient (hereafter also called prediction coefficient as appropriate).
Now, employing a linear combination model as the mapping method using this tap coefficient, (the pixel value) y of a pixel making up the HD image (hereafter referred to as HD pixel as appropriate) is obtained with the following linear expression (linear combination) using multiple SD pixels extracted from pixels making up the SD image, hereafter referred to as SD pixel as appropriate) as a prediction tap for predicting the HD pixel, and the tap coefficient.
                    y        =                              ∑                          n              =              1                        N                    ⁢                                    w              n                        ⁢                          x              n                                                          (        1        )            
Wherein, in Expression (1), xn represents the pixel value of the n'th pixel in the SD image, making up a prediction tap regarding the HD pixel y, and wn represents the n'th tap coefficient to be multiplied by (the pixel value of) the n'th SD pixel. Note that in Expression (1), a prediction tap is made up of an N number of SD pixels x1, x2, . . . , xN.
Now, the pixel value y of the HD pixel can be obtained by a quadratic expression or higher, instead of the linear expression shown in Expression (1).
Now, saying that yk represents the true value of the pixel value of an HD pixel in a k'th sample, and yk′ represents a prediction value of the true value yk obtained by the Expression (1), the prediction error ek thereof is as expressed in the following Expression.ek=yk−yk′  (2)
The prediction value yk′ in Expression (2) is obtained according to Expression (1), so substituting the yk′ in Expression (2) according Expression (1) yields the following Expression.
                              e          k                =                              y            k                    -                      (                                          ∑                                  n                  =                  1                                N                            ⁢                                                w                  n                                ⁢                                  x                                      n                    ,                    k                                                                        )                                              (        3        )            
Wherein, in Expression (3), Xn,k represents the n'th SD pixel making up the prediction tap for the HD pixel in the k'th sample.
While a tap coefficient wn wherein the prediction error ek is 0 in Expression (3) is optimal for predicting the HD pixel, obtaining such a tap coefficient wn for all HD pixels is generally difficult.
Accordingly, as a rule representing the tap coefficient wn as being optimal, employing the least-square method for example, means that the optimal tap coefficient wn can be obtained by minimizing the summation E of squared errors represented in the following Expression for example, as a statistical error.
                    E        =                              ∑                          k              =              1                        K                    ⁢                      e            k            2                                              (        4        )            
Wherein, in Expression (4), K represents the number of samples of sets made up of an HD pixel yk and SD pixels X1,k, X2,k, . . . , XN,k making up a prediction tap regarding that HD pixel yk.
The tap coefficient Wn which makes the summation E of squared errors in Expression (4) smallest (the minimum) is such that the partial differentiation of the summation E by the tap coefficient wn yields 0. Accordingly, the following Expression must be satisfied.
                                          ∂            E                                ∂                          w              n                                      =                                                            e                1                            ⁢                                                ∂                                      e                    1                                                                    ∂                                      w                    n                                                                        +                                          e                2                            ⁢                                                ∂                                      e                    2                                                                    ∂                                      w                    n                                                                        +            …            +                                          e                k                            ⁢                                                ∂                                      e                                          k                      ⁢                                                                                          ⁢                      2                                                                                        ∂                                      w                    n                                                                                =                      0            ⁢                                                  ⁢                          (                                                n                  =                  1                                ,                2                ,                …                ⁢                                                                  ,                N                            )                                                          (        5        )            
Now, partial differentiation of the above Expression (3) by the tap coefficient wn yields the following Expression.
                                                        ∂                              e                k                                                    ∂                              w                1                                              =                                                    -                                  x                                      1                    ,                    k                    ,                                                              ⁢                                                ∂                                      e                    k                                                                    ∂                                      w                    2                                                                        =                          -                              x                                  2                  ,                  k                                                                    ,        …        ⁢                                  ,                                            ∂                              e                                  k                  ⁢                                                                                                                            ∂                              w                N                                              =                                    -                              x                                  N                  ,                                                                          ⁢                  k                  ,                                                      ⁢                                                  ⁢                          (                                                k                  =                  1                                ,                2                ,                …                ⁢                                                                  ,                K                            )                                                          (        6        )            
Expressions (5) and (6) yield the following Expression.
                                                        ∑                              k                =                1                            k                        ⁢                                          e                k                            ⁢                              x                                  1                  ,                  k                                                              =          0                ,                                            ∑                              k                =                1                            k                        ⁢                                          e                k                            ⁢                              x                                  2                  ,                  k                                                              =          0                ,                              …            ⁢                                                  ⁢                                          ∑                                  k                  =                  1                                k                            ⁢                                                e                  k                                ⁢                                  x                                      N                    ,                    k                                                                                =          0                                    (        7        )            
Substituting Expression (3) for ek in Expression (7) allows Expression (7) to be expressed in the form of the normal equation in Expression (8).
                                          [                                                                                (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    …                                                                      (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      1                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                                                                    (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    ⋯                                                                      (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      2                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                                                ⋮                                                  ⋮                                                  ⋰                                                  ⋮                                                                                                  (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      1                            ,                            k                                                                                                                )                                                                                        (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      2                            ,                            k                                                                                                                )                                                                    …                                                                      (                                                                  ∑                                                  k                          =                          1                                                k                                            ⁢                                                                        x                                                      N                            ,                            k                                                                          ⁢                                                  x                                                      N                            ,                            k                                                                                                                )                                                                        ]                    ⁡                      [                                                                                w                    1                                                                                                                    w                    2                                                                                                ⋮                                                                                                  w                    N                                                                        ]                          ⁢                                                                                                                                                                                                      =                                                                                                                                =                                                                                                                                                                                                                                                                                                                                                  =                                              ⁢                                          [                                                                      (                                                            ∑                                              k                        =                        1                                            k                                        ⁢                                                                  x                                                  1                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                                                                      (                                                            ∑                                              k                        =                        1                                            k                                        ⁢                                                                  x                                                  2                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                                                    ⋮                                                                                      (                                                            ∑                                              k                        =                        1                                            k                                        ⁢                                                                  x                                                  N                          ,                          k                                                                    ⁢                                              y                        k                                                                              )                                                              ]                                    (        8        )            
Preparing a certain number of sets of the HD pixel yk and SD pixels xn,k allows as many of the normal equations in Expression (8) to be formulated as the number of tap coefficients wn to be obtained. Accordingly, solving Expression (8) (it should be noted that the matrices on the left side regarding the tap coefficient wn must be regular in Expression (8) in order to solve Expression (8)) allows optimal tap coefficients wn to be obtained. Note that sweeping (Gauss-Jordan elimination), for example, can be used for solving Expression (8).
As described above, adaptation processing involves taking a great number of HD pixels y1, y2, . . . , yK as tutor data to serve as a tutor for learning for tap coefficients, and also taking SD pixels X1,k, X2,k, . . . , XN,k making up a prediction tap regarding each HD pixel yk as student data to serve as a student for learning for tap coefficients, and solving Expression (8), thereby performing learning for obtaining an optimal tap coefficient wn, and further using the tap coefficient wn to perform mapping (conversion) of an SD image into an HD image according to Expression (1).
Hereafter, a tap coefficient will also be called a prediction coefficient.
Note that adaptation processing differs from simple interpolation processing or the like, for example, in that components not included in the SD image but are included in the HD image are reproduced. That is to say, with adaptation processing, it would seem by looking at Expression (1) alone that this is the same as interpolation processing using a so-called interpolation filter, but the tap coefficient wn which is equivalent to the tap coefficient of the interpolation filter has been obtained by learning using an HD image serving as tutor data and an SD image serving as student data, so the component contained in the HD image can be reproduced. Thus, adaptation processing can be said to be processing which as image creation (resolution imagination) effects, so as to say.
Now, tap coefficients wn which perform various types of conversion can be obtained in the tap coefficient wn learning, depending on what sort of combination of tutor data y and student data x is employed.
That is to say, in a case of taking an HD image as the tutor data y and taking an SD image wherein noise or blurring is added to this HD image as the student data x, for example, a tap coefficient wn for converting an image into an image with the noise and blurring thereof removed, can be obtained. Also, in a case of taking an HD image as the tutor data y and taking an SD image wherein the resolution of the HD image has been deteriorated as the student data x, for example, a tap coefficient wn for converting an image into an image with the resolution improved, can be obtained. Further, in a case of taking an image as the tutor data y and taking a DCT (Discrete Cosine Transform) coefficient obtained by DCT conversion of the image as the student data x, for example, a tap coefficient wn for converting a DCT coefficient into an image, can be obtained.
Next, description will be made regarding the configuration of a conventional image processing device for performing class classification adaptation processing.
FIG. 1 is a block diagram for describing the configuration of a conventional image processing device which creates an output image which is an HD image, from an input image which is an SD image, by class classification adaptation processing.
With the image processing device of which the configuration is shown in FIG. 1, the input image is supplied to a class tap extracting unit 11 and prediction tap extracting unit 15.
The class tap extracting unit 11 extracts, from the input image, a class tap which is predetermined pixels corresponding to a pixel which is of interest (hereafter also referred to as pixel of interest), and supplies the extracted class tap to a features detecting unit 12 along with the input image. The features detecting unit 12 detects features of the image corresponding to the pixel of interest from the input image supplied via the class tap extracting unit 11, and supplies the detected features to a class classification unit 13 along with the class tap. Note that features of an image means movement or in-frame change in pixel values.
The class classification unit 13 performs classification of the pixel of interest, based on the class tap and features supplied from the features detecting unit 12, and supplies a class code representing the classification result to coefficient memory 14 and the prediction tap extracting unit 15.
The coefficient memory 14 supplies a tap coefficient corresponding to the class of the pixel of interest to a pixel value computing unit 16, based on the class code supplied from the class classification unit 13.
The prediction tap extracting unit 15 extracts a predetermined prediction tap from the input image, corresponding to the pixel of interest, based on the class code supplied from the class classification unit 13. The prediction tap extracting unit 15 supplies the extracted prediction tap to the pixel value computing unit 16.
The pixel value predicting unit 16 predicts the pixel value of the pixel of interest in the HD image by the computation shown in Expression (1), from the prediction tap supplied from the prediction tap extracting unit 15 and the tap coefficient supplied from the coefficient memory 14. The pixel value predicting unit 16 outputs an HD image made up of pixel values predicted by sequentially taking all pixels in the HD image as a pixel of interest using class classification adaptation processing.
FIG. 2 is a flowchart describing the processing for creating an image with a conventional image processing device which creates an output image which is an HD image, from an input image which is an SD image.
In step S11, the class tap extracting unit 11 extracts a class tap corresponding to the selected pixel of interest, from the input image which is an SD image. In step S12, the features detecting unit 12 detects features corresponding to the pixel of interest, from the input image.
In step S13, the class classification unit 13 classifies the class of the pixel of interest, based on the class tap extracted by the processing in step S11 and the features detected by the processing in step S12.
In step S14, the prediction tap extracting unit 15 extracts a prediction tap corresponding top the pixel of interest from the input image, corresponding to the results of class classification by the processing in step S13. In step S15, the coefficient memory 14 reads out the prediction coefficient corresponding to the classified class, from prediction coefficients which are stored beforehand, corresponding to the class classification results of the processing in step S13.
In step S16, the pixel predicting unit 16 predicts the pixel value corresponding to the pixel of interest by adaptation processing, based on the prediction tap extracted in the processing in step S14 and the prediction coefficient read out in the processing in step S15.
In step S17, the image processing device determines whether or not prediction has ended for all pixels, and in the event that determination is made that prediction has not ended for all pixels, the flow returns to step S11 with the next pixel as the pixel of interest, and the class classification and adaptation processing is repeated.
In the event that determination is made in step S17 that prediction has been completed for all pixels, the processing ends.
FIG. 3 is a block diagram for describing the configuration of a conventional image processing device which generates prediction coefficients used in the class classification adaptation processing for creating an output image which is an HD image from an input image which is an SD image.
The input image which is input to the image processing device shown in FIG. 3 is a tutor image which is an HD image, and is supplied to a student image generating unit 31 and a tutor pixel extracting unit 38. (Pixels value of) pixels contained in the tutor image are used as tutor data.
The student image generating unit 31 generates a student image which is an SD image corresponding to the tutor image, by thinning out pixels form the input tutor image which is an HD image, and supplies the generated student image to image memory 32.
The image memory 32 stores the student image which is the SD image supplied from the student image generating unit 31, and supplies the stored student image to a class tap extracting unit 33 and prediction tap extracting unit 36.
The class tap extracting unit 33 sequentially selects the pixel of interest, extracts a class tap from the student image corresponding to the selected pixel of interest, and supplies the extracted class tap to a features detecting unit 34 along with the student image. The features detecting unit 34 detects the features from the student image, corresponding to the pixel of interest, and supplies the detected features to a class classification unit 35 along with the class tap.
The class classification unit 35 classifies the class of the pixel of interest based on the class tap and features supplied from the features detecting unit 34, and supplies a class code indicating the classified class to a prediction tap extracting unit 36 and learning memory 39.
The prediction tap extracting unit 36 extracts a prediction tap corresponding to the classified class from the student image supplied from the image memory 32, based on the class code supplied from the class classification unit 35, and supplies the extracted prediction tap to a supplementing computing unit 37.
A tutor pixel extracting unit 38 extracts the tutor data, i.e., the pixel of interest of the tutor image, and supplies the extracted tutor data to the supplementing computing unit 37.
The supplementing computing unit 37 supplements tutor data which is an HD pixel and the prediction tap which is SD pixels to the normal equation in Expression (8), and supplies the normal equation supplemented with the tutor data and prediction tap to the learning memory 39.
The learning memory 39 stores the normal equations supplied from the supplementing computing unit 37 by class, based on the class code supplied from the class classification unit 35. The learning memory 39 supplies the normal equations to which the tutor data and prediction tap has been supplemented, and stored for each class, to a normal equation computing unit 40.
The normal equation computing unit 40 solves the normal equations supplied from the learning memory 39 by sweeping, and obtains a prediction coefficient for each class. The normal equation computing unit 40 supplies the prediction coefficients for each class to coefficient memory 41.
The coefficient memory 41 stores the prediction coefficients for each class, supplied from the normal equation computing unit 40.
FIG. 4 is a flowchart for describing the processing of learning of the conventional image processing device which generates prediction coefficients used for class classification adaptation processing for creating an output image which is an HD image from an input image which is an SD image.
In step S31, the student image generating unit 31 generates a student image from an input image which is a tutor image. In step S32, the class tap extracting unit 33 sequentially selects the pixel of interest, and extracts a class tap corresponding to the selected pixel of interest from the student image.
In step S33, the features detecting unit 34 detects the features corresponding to the pixel of interest, from the student image. In step S34, the class classification unit 35 classifies the class of the pixel of interest based on the class tap extracted by the processing in step S32 and the features detected by the processing in step S33.
In step S35, the prediction tap extracting unit 36 extracts a prediction tap corresponding to the pixel of interest from the student image, based on the class classified by the processing in step S34.
In step S36, the tutor pixel extracting unit 38 extracts a pixel of interest from the input image which is a tutor image, i.e., extracts a tutor pixel (tutor data).
In step S37, the supplementing computing unit 37 performs computation for supplementing the prediction tap extracted in the processing in step S35 and the tutor pixel (tutor data) extracted in the processing in step S36 into the normal equation.
In step S38, the image processing device determines whether or not supplementing processing has ended for all pixels in the tutor image, and in the event that determination is made that the supplementing processing has not ended for all pixels, the flow returns to step S32, and the processing wherein a pixel which has not yet been taken as a pixel of interest is taken as the pixel of interest, the prediction tap and tutor pixel are extracted, and supplemented to the normal equation, is repeated.
In the event that determination is made in step S38 that the supplementing processing has ended for all pixels of the tutor image, the flow proceeds to step S39, where the normal equation computing unit 40 computes the normal equations where the prediction taps and tutor pixels have been supplemented, thereby obtaining prediction coefficients.
In step S40, the image processing device determines whether or not prediction coefficients have been calculated for all classes, and in the event that determination is made that prediction coefficients have not been calculated for all classes, the flow returns to step S39, and the processing for a calculating normal equation to obtain a prediction coefficient is repeated.
In the event that determination is made in step S40 that prediction coefficients have been calculated for all classes, the processing ends.
Also, Japanese Unexamined Patent Application Publication No. 9-74543 discloses not only class classification of input images based on level distribution patterns of input images, but also class classification of input images based on movement of input images to generate class codes.
However, image processing taking into consideration the data continuity of image data has not been heretofore conceived.